Android: custom Vulkan driver packages (Turnip) on Adreno - #19955

Sideloaded arm64 Android builds can load a Vulkan driver package, such as Mesa's Turnip, on Adreno GPUs instead of the system driver. The main use is cores the stock driver renders badly. On an Adreno 740, for example, the stock driver draws stripe garbage for paraLLEl-RDP in Mupen64Plus-Next (SM64, OoT, MK64), and T30, a Turnip build, renders all three correctly.

Related feature requests: #15472, #18143.

Commits

1. Android: load custom Vulkan driver packages on Adreno. Adds a new setting, video_vulkan_driver_path, that names a zip holding meta.json and one .so. An empty value means the system driver, and overrides carry it. The package is checked and extracted to internal storage once per version. This commit also adds the fallbacks, the crash state and the Safe start shortcut. The package logic is platform-free (gfx/common/vulkan_custom_driver.c), and a unit sample and a CI job cover it.
2. Menu: Custom Vulkan Driver setting. Adds a row under Settings > Video > Output. OK browses <system>/vulkan_drivers, and Start goes back to the system driver. The sublabel says when the chosen driver crashed. A new choice takes effect at the next Vulkan start, and picking one never restarts video.

The driver loading is a C port of libadrenotools' technique, in gfx/common/vulkan_custom_driver_linker.c, which keeps Billy Laws' BSD-2-Clause notice. A private copy of the system Vulkan loader is opened in a linker namespace of its own, where a small hook library hands it the custom driver. The file is built into libretroarch-activity.so and, a second time, as that hook, libvulkan_driver_hook.so. There is no C++ and no deps/ tree.

No driver is bundled. Users download a package (for example a Turnip build, in the same zip format the AdrenoTools-based emulators load) and copy it to <system>/vulkan_drivers/. If wanted, an online updater source can follow in a later PR.

Gating

• Build: arm64-v8a with Vulkan only, and all new code is under HAVE_VULKAN_CUSTOM_DRIVER. The API 16 build passes HAVE_VULKAN_CUSTOM_DRIVER=0.
• Play Store and Quest builds are excluded. Play's Device and Network Abuse policy forbids loading executable code (such as .so files) from outside Play. That is why the Play builds already get cores through Play Feature Delivery. The Quest build is also a Play build (PLAY_STORE_BUILD=1). On Quest, the OpenXR runtime also does swapchain interop on RetroArch's VkDevice, which nobody has tested with Turnip. Android.mk sets HAVE_VULKAN_CUSTOM_DRIVER only when neither PLAY_STORE_BUILD nor HAVE_OPENXR is set, and the Play and Quest APKs carry no custom driver symbols.
• Runtime: needs an Adreno GPU (/dev/kgsl-3d0) and Android 9 (API 28) or later. Elsewhere the row is hidden, and a synced-in value is ignored with an OSD message.
• The namespace setup, which remaps dlopen's code page to find the linker's internal dlopen, runs only when a custom driver is first opened, and libvulkan_driver_hook.so loads only then. Neither happens for users who don't pick a driver.

Fallbacks and crash state

Each case shows an OSD message once per path per session. A crash after the first swapchain is not caught. For that, the user presses Start on the row or uses Safe start.

Safe start is a launcher shortcut (Android 7.1+), added before the first custom load. It starts RetroArch once on the system driver, restarting the process if needed, and leaves the saved value alone. Meanwhile the row reads "System (Safe start)".

Vendor policy (please weigh in)

Packages from any vendor load. Only Mesa Turnip was tested. The non-Mesa path was checked with a Turnip build relabelled "vendor": "Qualcomm".

How it works:

• vendor is required in meta.json. A package counts as Mesa when it says "Mesa" (any case).
• Mesa packages get a load check. After the instance is created, the GPU's driverID must not be Qualcomm's proprietary one. If it is, the hook fell back to the stock driver without saying so. The start then counts as failed and the context is redone on the system driver.
• Other vendors only need a visible GPU. Once per init they log that the vendor is untested and that the load can't be confirmed.

Why: Qualcomm driver repacks are widely used with other Android emulators, so refusing them outright would turn those users away. But a repack reports the same driverID as the stock driver. The Mesa check would mark every repack as failed, because the driverID can't tell a working repack from a silent fallback. We also considered comparing against the stock driver's version, saved from an earlier system-driver run. That needs a prior run and more saved state, and it misfires when a repack has the same version as the stock driver. So other vendors load unverified, and the log says so rather than claiming the custom driver is running.

Qualcomm repacks are proprietary blobs taken from other devices. If you'd rather not load them, Mesa-only is a small change to the package check.

Known issues

• T30 + Dolphin + threaded video: with threaded video on, Dolphin crashed in 3 of 18 video reinits on T30. There were no crashes in 20 reinits on the system driver or in 10 on T30 with threaded video off. It also crashed with every core on T30 and no switching, so this is the driver, not driver switching.
• CACHE_CONTEXT fallback: after a device or swapchain failure, a new custom instance under a cached context takes its own fallback path. Code review and unit tests cover it, but no device run did.
• DEFLATE bound: libretro-common's streaming DEFLATE extraction checks the header's uncompressed size only after all input is used. A zip that understates it can write ~1000x its compressed size to internal storage before failing. DEFLATE64 and LZMA already stop early. The fix touches all archive extraction, so it will come as its own small PR.

Trying it

The one case found so far that the stock driver gets wrong and a custom driver fixes is paraLLEl-RDP in Mupen64Plus-Next. To see it on an Adreno 740 (tested on the AYN Thor, stock driver 0676.53):

1. Download turnip_mrpurple_T30-toasted.adpkg.zip from T30's release and copy it, still zipped, to <system>/vulkan_drivers/. T30's meta.json asks for Android 11 (API 30) or later.
2. Set the video driver to vulkan. In Mupen64Plus-Next's core options, set mupen64plus-rdp-plugin and mupen64plus-rsp-plugin to parallel.
3. On the system driver, Super Mario 64, Ocarina of Time and Mario Kart 64 (USA) show a black screen or colour stripes, and logcat fills with AdrenoVK-0 shader compile and pipeline-create failures.
4. Pick the zip under Settings > Video > Output > Custom Vulkan Driver, then close content and load the game again. All three games render, and the log's [Vulkan] Using GPU #0 line names PurpleVK-public Adreno (TM) 740 instead of the stock Adreno (TM) 740.

Not every Turnip build fixes it. K11MCH1's v26.0.0 R8 lacks the 8/16-bit storage paraLLEl-RDP needs on this GPU: Mupen64Plus-Next crashes in retro_run, and parallel_n64, which renders on the stock driver, goes black. Older Adreno GPUs such as the Adreno 650 in #18143 lack that storage on the stock driver too, and paraLLEl-RDP shows its error screen there. Whether a Turnip build fixes those GPUs was not tested.

Beetle PSX HW, SwanStation, Azahar, Dolphin, Flycast, PPSSPP and parallel_n64 rendered the same on T30 and on the stock driver, so they work as regression checks but don't need a custom driver.

Testing

• Unit sample samples/gfx/vulkan_custom_driver: 352 checks covering the package checks, meta.json, the crash state and session limits. It passes on Linux under ASan+UBSan, and in a mingw-w64 build under wine. The new vulkan-custom-driver CI job runs it.
• Builds, each commit: Linux configure + make, and NDK for sideloaded arm64, Play, Quest and x86_64. Only sideloaded arm64 ships libvulkan_driver_hook.so. C89_BUILD=1 on touched objects and --disable-threads both pass. Local runs of the covering CI jobs fail only where master does.
• Fork CI (CI: custom Vulkan driver, adrenotools shim as one C file Davey-Hughes/RetroArch#33): of 161 checks, 159 pass and 1 is skipped. The other, libsmb2, fails the same way on master 92b11d9.

AYN Thor (Adreno 740, Android 13), with T30 and "Mesa Turnip driver v26.0.0 - R8". The comparison, the bad packages, pruning and Safe start through its intent extra were rerun on this branch's final commits, with the same results as on the previous push (aed6a3f). The switching, closing-content and Safe start UI checks ran on an earlier build, before the final review fixes and the C port.

• Against the proof of concept: 66 cases (8 cores, 22 games, 3 drivers). Every case gave the same driver, crash and alive result as lim2222's PoC rebased on the same master, and the 44 Turnip screenshot pairs show no render differences.
• Switching: per-core overrides flipped four cores between T30 and system for 10 cycles. That gave 80 inits, all on the expected driver, each driver mapped once, no crashes. The memory growth per cycle is smaller than in a system-only control and comes from core load/unload.
• Closing content: all 229 closes reinitialised video on the base driver.
• Bad packages: each fell back to the system driver:
  • a truncated header (refused);
  • a truncated body (load failure detected);
  • a core .so (no GPU);
  • a constructor that segfaults (first start dies, later starts skip it with a crash message once).
• Other vendor: T30 relabelled "vendor": "Qualcomm" loads and runs, logging both warnings once per init.
• Pruning: replacing a zip deletes its old extraction at the next start, and a removed zip's extraction goes too.
• Safe start and the row: cold start, mid-game restart, no restart on a second tap, Start reset and crash-note clearing all behave as described.
• Android 9 path: a test build forced onto the temp-file copy of the loader, which only Android 9 uses, ran T30 in Mupen64Plus-Next, PPSSPP and SwanStation, and all three rendered.
• The custom driver mode alone works, so libadrenotools' file redirect hook is left out.

Tested on Android 13 (AYN Thor, Adreno 740). Android 9 takes a different path (the loader copy goes through a temporary file instead of a memfd); it would be good if someone with an Android 9 Adreno device could test it.

Credit

Built from lim2222's proof of concept, https://github.com/lim2222/RetroArch-Turnip (branch feature/android-turnip-driver). lim2222 is co-author of both commits.